Here's a scenario for you:
You need to restore a Domain Controller (DC), running Server 2012, back to a Hypervisor for testing. This might be via a SureBackup or in a disaster recovery situation using Veeam Backup and replication. However, upon successful restoration of the Virtual Machine files you encounter any of the following...
Symptoms
- The DC starts and initialises, but after a few minutes you receive a message stating that the computer needs to shut down. This behaviour continues in a loop.
- The DC starts but key services start to fail thereafter, such as DNS
- The DC presents a Blue Screen of Death (BSOD) on boot with error code 0XC00002E2 (a little something like the image below).
In all these circumstances you find that starting in DSRM allows the computer with the DC role to start and remain stable.
Cause
Veeam's Change Block Tracking driver (CBT) has failed to take a consistent copy of the active directory database correctly (NTDS.dit) and has corrupted the NTDS.dit within the Veeam backup of the DC. Although this is the case the Veeam console reports that the backup was taken successfully.
This is currently a known bug which Veeam are working on.
Resolution
To resolve this issue, disable CBT from the backup job which contains the DC. If the DC is part of a collection of virtual servers in a single job, it may be worth creating a separate Job for the DC with CBT disabled. This will allow leaving CBT enabled on the existing job to allow for faster incremental backups of the other servers.
Please note:- If the NTDS.dit is already corrupted, disabling CBT in the current job will not fix the corruption. It will also significantly increase the time taken to complete incremental backups.
A new job/chain will need to be created. If you have already started a new job due to recent backup failures problems CBT does not run on the first backup job so you can just disable it and continue.
More information
To test the consistency of the NTDS.dit in your current backup follow the following procedure.
Use Veeam to perform a file level restore of the NTDS folder from the DC: e.g. C:WindowsNTDS (This will depend on where you have selected to create this folder.) This folder need to be copied to a location on the DC i.e C:Temp
Open up an administrative command prompt and type Esentutl /g
e.g. esentutl /g C:tempNTDSntds.dit
The Esentutl tool will confirm whether AD/NTDS is corrupted.
Please note:- This does not mean that your live AD/NTDS.dit on the DC is corrupted. This just shows that the AD/NTDS.dit within the backup has been corrupted due to CBT. You can confirm this by looking at event viewer in the following areas. You can also perform the same integrity check on the live ntds.dit by using one of the following methods.
Method 1
- Stop the Active Directory Domain services. (Warning this stops AD from functioning in the live environment). This will need to be planned downtime.
- Open an elevated command prompt and type Esentutl /g C:WindowsNTDSNTDS.dit (change your path accordingly)
- Start the Active Directory Domain Services
Method 2
- From an elevated command prompt type the following Diskshadow
- Set Verbose on - gives some extra output
- Add volume C - this adds the specified volume to the shadow copy set, if you want another volume, type one more command e.g. Add volume D
- Set Context Persistent - required parameter for shadow copy if you plan to mount it. You will need to delete the snapshot manually afterwards
- Create - starts process of creation, when it finishes you will have shadow copy ID
- Expose shadow copy ID X: (or any other letter) - mounts a shadow copy as a drive letter specified
- Exit Diskshadow - Exits Diskshadow
- At an elevated Command prompt type Esentutl /g X:WindowsNTDSNTDS.dit (change your path depending on the drive letter of your mounted snapshot)
- Once finished type Diskshadow
- Delete shadows exposed x: (change your path depending on the drive letter of your mounted snapshot) - dismounts and deletes shadow copy
More info about these commands can be found here: http://technet.microsoft.com/en-us/library/cc772172.aspx
